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{f## Status: Path 1 of [Dialog Information Services via Modem] 

### Status: Initializing TCP/IP using (UseTelnetProto 1 ServicelD pto-dialog) 
Trying 3106900061 ... Open 

DIALOG INFORMATION SERVICES 
PLEASE LOGON: 

******** HHHHHHHH SSSSSSSS? 
### Status: Signing onto Dialog 

ENTER PASSWORD: 

******** HHHHHHHH SSSSSSSS? ******** 
Welcome to DIALOG 
### Status: Connected 



Dialog level 00. 06. SOD 

Last logoff: 20jun00 09:28:13 
Logon fileOOl 22jun00 08:55:30 
*** ANNOUNCEMENT *** 
NEW FILE RELEASED 

***Prous Science Daily Essentials (Files 458, 459) 
***WIPO/PCT Patents Fulltext (File 349) 



UPDATING RESUMED 

***Bridge World Markets News (File 609,809) 
***Fort Worth Star-Telegram (File 427) 
***Federal News Service (File 660) 
***Kansas City Star (File 147) 
***British Books in Print (File 430) 



RELOADED 

***Books in Print (File 470) 
***Kompass Asia/Pacific (File 592) 

»>Get immediate news with Dialog's First Release 

news service. First Release updates major newswire 
databases within 15 minutes of transmission over the 
wire. First Release provides full Dialog searchability 
and full-text features. To search First Release files in 
OneSearch simply BEGIN FIRST for coverage from Dialog's 
broad spectrum of news wires. 

»> Enter BEGIN HOMEBASE for Dialog Announcements «< 
»> of new databases, price changes, etc. «< 
* ★ + + 



File 1:ERIC 1966-2000/ Jun 17 

(c) format only 2000 The Dialog Corporation 
*File 1: File has been reloaded. See HELP NEWS 1. 



Set Items Description 



Frantz Coby 




06/22/00 




2 



?begin 411 

22junOO 08:55:42 User219455 Session D638.1 
$0.22 0.062 DialUnits Filel 
$0.22 Estimated cost Filel 
$0.01 TYMNET 

$0.23 Estimated cost this search 

$0.23 Estimated total session cost 0.062 DialUnits 
File 411:DIALINDEX(R) 

DIALINDEX(R) 

(c) 2000 The Dialog Corporation pic 

*** DIALINDEX search results display in an abbreviated *** 

*** format unless you enter the SET DETAIL ON command. *** 

?sf compsci, patents 

»> 64 does not exist 

>» 351 is unauthorized 

»> 352 is unauthorized 

»> 353 is unauthorized 

»>4 of the specified files are not available 

You have 36 files in your file list. 

(To see banners, use SHOW FILES command) 
?show files 

File Name 



2: INSPEC_1969-2000/May W2 
6: NTIS_1964-2000/Jul W2 
8: Ei Compendex(R)_1970-2000/May W4 
34: SciSearch(R) Cited Ref Sci_1990-2000/ Jun W3 
35: DISSERTATION ABSTRACTS ONLINE_1861-1999/DEC 
65: Inside Conf erences_1993-2000/ Jun W3 
77: Conference Papers Index_1973-2000/May 
92: IHS Intl.Stds.& Specs . _1999/Nov 
94: JICST-EPlus_1985-2000/Feb W3 
99: Wilson Appl . Sci & Tech Abs_1983-2000/May 

103: Energy SciTec_1974-2000/Apr Bl 

108: Aerospace Database_1962-2000/May 

144: Pascal_1973-2000/Jun W3 

202: Information Science Abs ._1966-2000/Mar 

233: Internet & Personal Comp. Abs ,_1981-2000/ Jun 

238: Abs. in New Tech & Eng . _1981-2000/ Jun 

239: Mathsci_1940-2000/Jul 

275: Gale Group Computer DB (TM) _1983-2000/ Jun 22 

434: SciSearch{R) Cited Ref Sci_1974-1989/Dec 

647: CMP Computer Fulltext_1988-2000/ Jun Wl 

674: Computer News Fulltext_1989-2000/May W4 

696: DIALOG Telecom. Newsletters_1995-2000/ Jun 21 

123: CLAIMS (R) /Current Legal Status_1980-2000/ Jun 06 

340: CLAIMS (R) /US Patent_1950-00/ Jun 13 

342: Derwent Patents Citation Indx_1978-98/200005 

344: Chinese Patents ABS_Apr 1985-2000/Feb 

345: Inpadoc/Fam, & Legal Stat_1968-2000/UD=200023 

347: JAPIO_Oct 197 6-2000/ Jan (UPDATED 000611) 

348: European Patents_1978-2000/ Jun W03 

349: PCT Fulltext 1983-2000/UB=, UT=20000525 



Frantz Coby 



06/22/00 



3 



371: French Patents_1961-2000/BOPI 0023 

447: IMSWorld Patents International_2000/May 

652: US Patents Fulltext_1971-1979 

653: US Patents Fulltext_1980-1989 

654: US Pat. Full. _1990-2000/Jun 20 

670: LitAlert_1973-2000/UD=200019 

?s temporal databases and hyperlink and ({time or temporal (w) descriptor? ) or 
date) 

Your SELECT statement is: 

s temporal databases and hyperlink and ((time or temporal (w) descriptor? ) 

or date) 

Items File 

1 2: INSPEC_1969-2000/May W2 

1 file has one or more items; file list includes 36 files. 
?begin 2 

22jun00 08:58:00 User219455 Session D638.2 
$0.56 0.450 DialUnits File411 
$0.56 Estimated cost File411 
$0.15 TYMNET 

$0.71 Estimated cost this search 

$0.94 Estimated total session cost 0.513 DialUnits 



File 2:INSPEC 1969-2000/May W2 

(c) 2000 Institution of Electrical Engineers 



Set Items Description 

?s temporal databases and hyperlink and ({time or temporal (w) descriptor? ) or 
date) 

1077 TEMPORAL DATABASES (January 1993) 

125 HYPERLINK 

926723 TIME 

55001 TEMPORAL 

4586 DESCRIPTOR? 

4 TEMPORAL (W) DESCRIPTOR? 

27188 DATE 

SI 1 TEMPORAL DATABASES AND HYPERLINK AND ((TIME OR 

TEMPORAL (W) DESCRIPTOR?) OR DATE) 

?t 1/9/1 



1/9/1 

DIALOG (R) File 2:INSPEC 

{c) 2000 Institution of Electrical Engineers. All rts . reserv. 

4999824 INSPEC Abstract Number: C9509-6130M-006 
Title: Multimedia interchange using SGML/HyTime. I. Structures 
Author(s): Newcomb, S.R. 

Journal: IEEE Multimedia vol.2, no. 2 p. 86-9 
Publication Date: Summer 1995 Country of Publication: USA 
ISSN: 1070-986X 
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U.S. Copyright Clearance Center Code: 1070-986X/95/$4 . 00 
Language: English Document Type: Journal Paper (JP) 
Treatment: Practical (P) 

Abstract: HyTime is a standard neutral markup language for representing 
hypertext, multimedia, hypermedia and time -based documents in terms of 
their logical structure. Documents represented in HyTime conform fully to 
the ISO Standard Generalized Markup Language (SGML) . In effect, HyTime 
extends SGML by adding certain syntactic conventions called SGML 
architectural forms, with which it represents certain semantic constructs. 
HyTime cannot be understood or evaluated without understanding, at least to 
some extent, the significance and usefulness of the SGML standard on which 
it is based. This article examines the structure of the two standards. 
HyTime places unprecedented demands on document processing systems -demands 
which they have yet to meet. For example, a full implementation of HyTime 
would allow one to create a hyperlink to whatever happens to be going on 
at a particular time and/or place in a finite coordinate space (FCS) , 
even if the event, location and time are not yet known, because of 
binding- time issues. HyTime allows a document to specify where and/or 
when the results of traversing a hyperlink will be rendered. HyTime 
provides constructs for specifying how events scheduled in one FCS are to 
be "projected" onto another, e.g. from a 3D FCS to a 2D FCS, or from a 
virtual measurement domain to a real one. (2 Refs) 

Descriptors: data structures; electronic data interchange; hypermedia; 
ISO standards; multimedia computing; page description languages; software 
packages; software standards; temporal databases 

Identifiers: multimedia interchange; semantic constructs; HyTime; 
standard neutral markup language; hypertext; multimedia; hypermedia; time 
-based documents; logical structure; ISO Standard Generalized Markup 
Language; syntactic conventions; SGML architectural forms; document 
processing systems; hyperlink traversal; finite coordinate space; 
binding-time issues; event scheduling; virtual measurement domain 

Class Codes: C6130M (Multimedia); C6130D (Document processing techniques) 
; C6140D (High level languages); C6160S (Spatial and pictorial databases); 
C6130E (Data interchange); C6120 (File organisation) 

Copyright 1995, lEE 
?begin 411 

22jun00 09:01:27 User219455 Session D638.3 
$1.18 0.196 DialUnits File2 

$2.10 1 Type(s) in Format 9 
$2.10 1 Types 
$3.28 Estimated cost File2 
$0.20 TYMNET 

$3.48 Estimated cost this search 

$4.42 Estimated total session cost 0.708 DialUnits 

File 411:DIALINDEX(R) 

DIALINDEX(R) 

(c) 2000 The Dialog Corporation pic 

*** DIALINDEX search results display in an abbreviated *** 

*** format unless you enter the SET DETAIL ON command. *** 

?sf compsci, patents 

»> 64 does not exist 

»> 351 is unauthorized 

»> 352 is unauthorized 

»> 353 is unauthorized 
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»>4 of the specified files are not available 

You have 36 files in your file list. 

(To see banners, use SHOW FILES command) 
?s temporal database? and hyperlink and ({time or temporal(w) descriptor? ) or 
date) 

Your SELECT statement is: 

s temporal database? and hyperlink and ((time or temporal (w) descriptor? ) 
or date) 

Items File 

1 2: INSPEC_1969-2000/May W2 

1 file has one or more items; file list includes 36 files. 
?s temporal (w) database? and ((time or temporal (w) descriptor? ) or date) 

Your SELECT statement is: 

s temporal (w) database? and ((time or temporal (w) descriptor? ) or date) 

Items File 



707 • 2: INSPEC_1969-2000/May W2 

18 6: NTIS_1964-2000/Jul W2 

128 8: Ei Compendex (R) _1970-2000/May W4 

101 34: SciSearch(R) Cited Ref Sci_1990-2000/ Jun W3 
55 35: DISSERTATION ABSTRACTS 0NLINE_18 61- 1999/DEC 
31 65: Inside Conf erences_1993-2000/ Jun W3 

1 77: Conference Papers Index_1973-2000/May 
10 94: JICST-EPlus_1985-2000/Feb W3 
17 99: Wilson Appl . Sci & Tech Abs_1983-2000/May 

8 103: Energy SciTec_1974-2000/Apr Bl 
10 108: Aerospace Database_1962-2000/May 

132 144: Pascal_1973-2000/Jun W3 

22 202: Information Science Abs . _1966-2000/Mar 

1 233: Internet & Personal Comp. Abs .__1981-2000/ Jun 

9 239: Mathsci_1940-2000/Jul 

24 275: Gale Group Computer DB (TM) _1983-2000/ Jun 22 

1 647: CMP Computer Fulltext_1988-2000/ Jun Wl 

2 340: CLAIMS (R) /US Patent_1950-00/ Jun 13 

1 345: Inpadoc/Fam. & Legal Stat_1968-2000/UD=200023 

3 348: European Patents_1978-2000/ Jun W03 

12 349: PCT Fulltext_1983-2000/UB=, UT=20000525 
24 654: US Pat . Full . _1990-2000/ Jun 20 

22 files have one or more items; file list includes 36 files. 

? s temporal (w) database? and ({time or temporal (w) descriptor? ) or date?) and 
(hyperlink or url) 

Your SELECT statement is: 

s temporal (w) database? and ((time or temporal (w) descriptor? ) or date?) 
and (hyperlink or url) 



Items File 
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1 2: INSPEC_1969-2000/May W2 

»>File 349 processing for DATE? stopped at DAT EN FRAGMENT E 

2 349: PCT Fulltext_1983-2000/UB=, UT=20000525 
1 654: US Pat. Full. _1990-2000/Jun 20 

3 files have one or more items; file list includes 36 files. 
One or more terms were invalid in 3 files. 

?begin 2,654 

22jun00 09:05:51 User219455 Session D638.4 
$2.61 2,085 DialUnits File411 
$2.61 Estimated cost File411 
$0.25 TYMNET 

$2.86 Estimated cost this search 

$7.28 Estimated total session cost 2.794 DialUnits 



SYSTEM: OS - DIALOG OneSearch 

File 2:INSPEC 1969-2000/May W2 

(c) 2000 Institution of Electrical Engineers 
File 654:US Pat. Full. 1990-2000/ Jun 20 

(c) format only 2000 The Dialog Corp. 
*File 654: Reassignment data current through 12/06/1999 recordings. 
Due to recent processing problems, the SORT command is not working. 



Set Items 


Description 


?s temporal (w) database? and ((time or temporal (w) descriptor? ) or date?) 


(hyperlink or url) 




70353 


TEMPORAL 


133239 


DATABASE? 


1177 


TEMPORAL (W) DATABASE? 


1775951 


TIME 


70353 


TEMPORAL 


9621 


DESCRIPTOR? 


4 


TEMPORAL (W) DESCRIPTOR? 


145341 


DATE? 


490 


HYPERLINK 


1797 


URL 


SI 2 


TEMPORAL (W) DATABASE? AND ( (TIME OR 




TEMPORAL (W) DESCRIPTOR?) OR DATE?) AND (HYPERLINK OR URL) 


?begin 349, 654 




22jun00 09: 


06:52 User219455 Session D638.5 


$0.44 


0.074 DialUnits File2 


$0.44 Estimated cost File2 


$0.66 


0.112 DialUnits File654 



$0.66 Estimated cost File654 

OneSearch, 2 files, 0.186 DialUnits FileOS 
$0.10 TYMNET 

$1.20 Estimated cost this search 

$8.48 Estimated total session cost 2.980 DialUnits 



SYSTEM: OS - DIALOG OneSearch 

File 349: PCT Fulltext 1983-2000/UB=, UT=20000525 

(c) 2000 WIPO/MicroPatent 
File 654 :US Pat. Full. 1990-2000/ Jun 20 
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(c) format only 2000 The Dialog Corp. 
*File 654: Reassignment data current through 12/06/1999 recordings. 
Due to recent processing problems, the SORT command is not working. 

Set Items Description 

?s temporal (w) database? and ((time or temporal (w) descriptor? ) or date?) and 
{hyperlink or url) 

>»File 349 processing for DATE? stopped at DAT EN FRAGMENT E 



22534 


TEMPORAL 




50057 


DATABASE? 




36 


TEMPORAL (W) DATABASE? 




1100141 


TIME 




22534 


TEMPORAL 




6881 


DESCRI PTOR? 




0 


TEMPORAL (W) DESCRIPTOR? 




163227 


DATE? 




627 


HYPERLINK 




3116 


URL 




3 


TEMPORAL (W) DATABASE? AND 


( (TIME OR 




TEMPORAL (W) DESCRIPTOR?) OR 


DATE?) AND (HYPERLINK OR URL) 



?t l/2,ab/l-3 

1/2,AB/1 (Item 1 from file: 349) 

DIALOG (R) File 349:PCT Fulltext 
(c) 2000 WIPO/MicroPatent . All rts . reserv. 

00697153 

MULTI-PERSPECTIVE VIEWER FOR CONTENT-BASED INTERACTIVITY 

VISUALISEUR MULTIDIMENSIONNEL OFFRANT UNE INTERACTIVITE ORIENTEE OBJET 
Patent Applicant/Assignee: 

PRAJA INC; Address - PRAJA, INC. , 10455-B Pacific Center Court, San 
Diego, CA 92121 , US 
Inventor (s ) : 

JAIN Ramesh C; Address - JAIN, Ramesh, C. , 4715 Reedley Terrace, San 

Diego, CA 92130 , US 
HICKS Terry Randolph; Address - HICKS, Terry, Randolph , 4240 Trias 

Street, San Diego, CA 92103 , US 
BAILEY Asquith A; Address - BAILEY, Asquith, A. , 12055 World Trade Drive 

&3, San Diego, CA 92128 , US 
MCKINLEY Ryan B; Address - MCKINLEY, Ryan, B. , 21 Santa Lucia, Orinda, 

CA 94563 , US 

KURAMURA Don Yamato; Address - KURAMURA, Don, Yamato , 10669 Indigo 

Blossom Lane, San Diego, CA 92121 , US 
KATKERE Arun L; Address - KATKERE, Arun, L. , 3899 Nobel Drive &1228, San 

Diego, CA 92122 , US 
Patent and Priority Information (Country, Number, Date) : 

Patent: (WO 200010075) WO 0010075 Al 20000224 

Application: WO 99US18376 19990812 (PCT/WO US9918376) 

Priority Application: US 98134188 19980814 
Designated States: AE; AL; AM; AT; AU; AZ; BA; BB; BG; BR; BY; CA; CH; CN; 
CU; CZ; DE; DK; EE; ES; FI; GB; GD; GE; GH; GM; HR; HU; ID; IL; IN; IS; 
JP; KE; KG; KP; KR; KZ; LC; LK; LR; LS; LT; LU; LV; MD; MG; MK; MN; MW; 
MX; NO; NZ; PL; PT; RO; RU; SD; SE; SG; SI; SK; SL; TJ; TM; TR; TT; UA; 
UG; UZ; VN; YU; ZA; ZW; GH; GM; KE; LS; MW; SD; SL; SZ; UG; ZW; AM; AZ; 
BY; KG; KZ; MD; RU; TJ; TM; AT; BE; CH; CY; DE; DK; ES; FI; FR; GB; GR; 
IE; IT; LU; MC; NL; PT; SE; BF; BJ; CF; CG; CI; CM; GA; GN; GW; ML; MR; 
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NE; SN; TD; TG 
Main International Patent Class: G06F-003/14; 
Publication Language: English 
Filing Language: English 
Fulltext Word Count: 20046 

English Abstract 

A method and apparatus for interactively viewing a real-world 
environment. The viewer (4 00) includes a user interface having a first 
window (406) for displaying a two-dimensional representation of a three- 
dimensional model of the real world environment. The viewer (400) further 
includes a plurality of other viewing areas (402-424) for displaying and 
querying for views of the real-world environment. The viewer (400) 
includes a content-based event timeline (422) that graphically depicts 
multi-media events satisfying user queries. Several methods can be used 
to select an event for display. For example, the user can select an event 
by selecting the event representation from the timeline (422) . 
Alternatively, the user can select an event by querying the system for 
selected objects, viewing angles, input devices, etc. The viewer (400) 
synchronizes and links together all of the multi-media data types 
associated with a selected event. Thus, when the user selects an event, 
the viewer (400) displays all of the multi-media information (such as 
audio/video and textual information) that is associated with selected 
event . 

French Abstract 

L* invention concerne un precede et un appareil pour visualiser de maniere 
interactive un environnement reel. Le visualiseur (400) inclut une 
interface utilisateur presentant une premiere fenetre (406) pour afficher 
une representation bidimensionnelle d*un modele tridimensionnel de 
1' environnement reel. Le visualiseur (400) inclut egalement une pluralite 
d'autres zones de visualisation (402 a 424) pour presenter et rechercher 
des vues de 1 ' environnement reel. Le visualiseur (400) inclut en outre un 
sequenceur d'evenements oriente objet (422) qui represente graphiquement 
des evenements multimedia pour satis f aire la demande d ' utilisateurs . 
Plusieurs methodes peuvent etre utilisees en ce qui concerne le choix 
d'un evenement a presenter. Par exemple, - 'utilisateur peut opter pour 
un evenement en choisissant la representation de 1 ' evenement a partir du 
sequenceur (422). Dans un mode different, 1 ' utilisateur peut choisir un 
evenement en inter rogeant le systeme pour des objets choisis, des angles 
de visualisation, des dispositifs d' entree, etc. Le visualiseur (400) 
sychronise et lie ensemble tous les types de donnees multimedia associes 
a 1* evenement choisi. Ainsi, lorsque 1 ' utilisateur choisit un evenement, 
le visualiseur (400) presente toutes les informations multimedia (telles 
que des informations audio/video ou textuelles) associees a 1* evenement 
choisi . 



l/2,AB/2 (Item 2 from file: 349) 

DIALOG (R) File 349:PCT Fulltext 
(c) 2000 WIPO/MicroPatent . All rts . reserv. 

00435633 

IMMERSIVE VIDEO 

VIDEO D' IMMERSION 

Patent Applicant/Assignee: 

THE REGENTS OF THE UNIVERSITY OF CALIFORNIA 
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JAIN Ramesh 
WAKIMOTO Koji 
MOEZZI Saied 
KATKERE Arun 
Inventor (s) : 
JAIN Ramesh 
WAKIMOTO Koji 
MOEZZI Saied 
KATKERE Arun 

Patent and Priority Information (Country, Number, Date) : 
Patent: WO 9631047 A2-A3 19961003 

Application: WO 96US4400 19960329 (PCT/WO US9604400) 

Priority Application: US 95414437 19950331; US 95554848 19951107 
Designated States: AL; AM; AT; AU; AZ; BB; BG; BR; BY; CA; CH; CN; CZ; DE; 

DK; EE; ES; FI ; GB; GE; HU; IS; JP; KE; KR; KZ; LK; LR; LS; LT; LU; LV; 

MD; MG; MK; MN; MW; MX; NO; NZ; PL; PT; RO; RU; SD; SE; SG; SI; TM; TR; 

TT; UA; UG; US; UZ; VN; KE; LS; MW; SD; SZ; UG; AM; AZ; BY; KG; KZ; MD; 

RU; TJ; TM; AT; DE; DK; ES; FI; FR; GB; GR; IE; IT; LU; MC; NL; PT; SE; 

BF; BJ; CF; CG; CI; CM; GA; GN; ML; MR; TD; TG 
Main International Patent Class: G06K-009/20; 
Publication Language: English 
Fulltext Word Count: 48912 

English Abstract 

Immersive video, or television, images of a real-world scene are 
synthesized (i) on demand, (ii) in real time , (iii) as linked to any of 
a particular perspective on the scene, or an object or event in the 
scene, (iv) in accordance with user-specified parameters of presentation, 
including panoramic or magnified presentations, and/or (v) 
stereoscopically. The synthesis of virtual images is based on 
computerized video processing — called "hypermosaicing" — of multiple 
live video perspectives on the scene. In hypermosaicing a knowledge 
database contains information about the scene; for example scene 
geometry, shapes and behaviors of objects in the scene, and/or internal 
and/or external camera calibration models. Multiple video cameras each at 
a different spatial location produce multiple two-dimensional video 
images of the scene. A viewer/user specifies viewing criterion (ia) at a 
viewer interface. A computer, typically one or more engineering work 
station class computers or better, includes in software and/or hardware 
(i) a video d^ita analyzer for detecting and for tracking scene objects 
and their locations, (ii) an environmental model builder combining 
multiple scene images to build a 3-D dynamic model recording scene 
objects and their instant spatial locations, (iii) a viewer criterion 
interpreter, and (iv) a visualizer for generating from the 3-D model in 
accordance with the viewing criterion one or more particular 2-D video 
image (s) of the scene. A video display receives and displays the 
synthesized 2-D video image{s). Nonetheless to being built and maintained 
by use of simplifying assumptions, the 3-D dynamic model is powerful, 
flexible and useful in permitting diverse scene views. 

French Abstract 

Des images video ou televisees d' immersion, d'une scene reelle, sont 
synthetisees (a) a la demande, (b) en temps reel, (c) parce qu ' etant 
liees a une perspective particuliere sur la scene, a un sujet ou a un 
evenement de la scene, (d) conformement a des parametres, specifies par 
I'utilisateur, de presentation, y compris les presentations panoramiques 
ou agrandies, et/ou (e) en stereoscopie . La synthese d' images virtuelles 



Frantz Coby 



06/22/00 



10 



est fondee sur le traitement video informatise - appele "hypermosaiquage" 
- de perspectives video multiples en direct sur la scene. Dans 
I'hypermosaiquage, une base de donnees de connaissances contient des 
informations sur la scene, par exemple ses caracteristiques geometriques, 
les formes et les comportements de sujets et/ou les modeles interieurs 
et/ou exterieurs de calibrage des cameras. Des cameras video multiples, 
chacune a un emplacement spatial different, produisent des images video 
multiples a deux dimensions de la scene. Un spectateur, ou utilisateur, 
specifie un ou plusieurs criteres de vision, sur une interface pour 
spectateur. Un ordinateur, habituellement un ou plusieurs ordinateurs de 
la categorie employee pour les postes de travail industriels, ou 
meilleurs que cela, comprend dans son logiciel et/ou dans sa partie 
materielle (a) un analyseur de donnees video, pour detecter et suivre les 
sujets de la scene et leurs emplacements, (b) un generateur de modeles 
d'environnement, qui combine des images de scene multiples pour generer 
un modele dynamique a trois dimensions enregistrant des sujets de la 
scene et leurs emplacements spatiaux instantanes, (c) un dispositif 
d' interpretation des criteres du spectateur, et (d) un dispositif de 
visualisation, pour produire a partir du modele a trois dimensions, 
conformement au critere de vision, une, ou davantage, image (s) video de 
la scene, en deux dimensions. Un visuel video recoit et montre les images 
video a deux dimensions qui ont ete synthetisees . Bien qu^etant elabore 
et maintenu grace a des hypotheses simplif icatrices , le modele dynamique 
a trois dimensions est puissant, souple et utile du fait qu'il permet 
d' avoir des vues di verses de la scene. 



l/2,AB/3 (Item 1 from file: 654) 

DIALOG (R) File 654: US Pat. Full. 

(c) format only 2000 The Dialog Corp. All rts . reserv. 
02888809 

Utility 

IMMERSIVE VIDEO, INCLUDING VIDEO HYPERMOSAICING TO GENERATE FROM MULTIPLE 
VIDEO VIEWS OF A SCENE A THREE-DIMENSIONAL VIDEO MOSAIC FROM WHICH DIVERSE 
VIRTUAL VIDEO SCENE IMAGES ARE SYNTHESIZED, INCLUDING PANORAMIC, SCENE 
INTERACTIVE AND STEREOSCOPIC IMAGES 

PATENT NO. : 5, 850,352 

ISSUED: December 15, 1998 (19981215) 

INVENTOR (s ) : Moezzi, Saied, San Diego, CA (California), US (United States 
of America) 

Katkere, Arun, La Jolla, CA (California), US (United States of 
America) 

Jain, Ramesh, San Diego, CA (California), US (United States of 
America) 

ASSIGNEE (s) : The Regents of the University of California, (A U.S. Company 
or Corporation), Alameda, CA (California), US (United States 
of America) 

[Assignee Code(s): 13234] 
APPL. NO. : 8-554, 848 

FILED: November 06, 1995 (19951106) 



REFERENCE TO A RELATED PATENT APPLICATION 



The present patent application is a continuation-in-part of U.S. patent 
application Ser. No. 08-414,437 filed on Mar. 31, 1995 to inventors Ramesh 



Frantz Coby 



oenmo 



11 



Jain and Koji Wakimoto for MACHINE DYNAMIC SELECTION OF ONE VIDEO 
CAMERA-IMAGE OF A SCENE FROM MULTIPLE VIDEO CAMERAS -IMAGES OF THE SCENE IN 
ACCORDANCE WITH A PARTICULAR PERSPECTIVE ON THE SCENE, AN OBJECT IN THE 
SCENE, OR AN EVENT IN THE SCENE. The contents of the related predecessor 
patent application are incorporated herein by reference. 

U.S. CLASS; 364-514 A cross ref: 348-13 
INTL CLASS: [6] H04N 9-79 

FIELD OF SEARCH: 364-514A; 364-410; 348-13; 348-19; 348-42; 348-51; 348-48; 

273-433; 273-441; 395-119; 395-125; 395-155; 395-129; 395-162 
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...images of a real-world scene are synthesized, including on demand and/or 
in real time , as are linked to any of a particular perspective on the 
scene, or an object... 
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. . . four cameras simultaneously recording the scene of a campus courtyard at 
a particular instant of time . 

FIG. 7 is four selected virtual camera, or synthetic video, images taken 
from a 116. . . 

...view of the same courtyard previously seen in FIG. 6 at the same instant 
of time . 

FIG. 9a is a graphical rendition of the 3D environment model generated 
for the same time instant shown in FIG. 6b, the volume of voxels in the 
model intentionally being at... by the environmental model builder of the 
immersive video system of the present invention, this time for an outdoor 
karate demonstration, this time the environmental model being further 
shown to be located in the static scene, particularly of ... diagrammatic 
representation of the terms of the algorithm, of the present invention 
where, at each time instant, multiple vistas are computed using the 
current dynamic model and video streams from multiple... 

...demand, (ii) the synthesis of virtual video images of a real-world scene 
in real time , or virtual television, (iii) the synthesis of virtual video 
images/ virtual television pictures of a... video. MPI video is the basis, 
and the core, of the "immersive video" (non-real-time and "immersive 
telecresence" or "Visualized Reality (VisR) (real-time ), systems of the 
present invention. The MPI Video system itself overcomes several 
limitations of the... a full custom virtual video image for each viewer is 
to be synthesized in real time and on demand requires that the model 
should be constructed ... A "next step" in MPI video beyond^ this 
rudimentary implementation is as a non-real-time pre-processed "game 
video". Such a "game video" may be recorded on the now-emerging... 

. . . will be to send the same information on twenty-three channels live, and 
in real time , on game day. Subscriber/viewer voting may permit a limited 
interaction. For example, the "fans... and what will undoubtedly transpire 
only after the lapse of some years from the present time (1995), it 
should be possible for each fan to be his or her own "game director", and 
to watch in real time substantially exactly what he or she wants. 

Accordingly, to exercise even the MPI video system. ..the MPI video system 
to realize untrammeled video views — including stereoscopic views. In 
non-real-time applications this realization, and the special processes of 
so realizing, are called "immersive videos. In real-time applications the 
realization, and the processes, are "immersive telepresence", or "visual 
reality", or "VisR". In. . .M. Hansen, P. Anandan, K. Dana, G. Van der Wal 
and P. Burt in Real-time scene stabilization and mosaic construction, 
appearing in ARPA Image Understanding Workshop, Monterey, Calif., Nov. 13 
... In particular, multiple independent streams of video data of the same 
scene at the same time will be seen to be input to the system. Because 
the system of the present ... see with two eyes, we do not see things from 
all directions at the same time . Humans have, or course, highly evolved 
brains, and perception. However, at least one situation of... OF THE 
INVENTION 

The present invention contemplates telepresence and immersive video, 
being the non-real-time creation of a synthesized, virtual, camera/video 
image of a real-world scene, typically in... of presentation is (are) 
normally specified by, and may be changed at times and from time to time 
by, a viewer/user of the system. Because the criterion (criteria) is (are) 
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changeable, the... cameras, and also the viewer-specified criterion 
(criteria) from the viewer interface. At the present time , circa 1995, 
the typical computer functioning in an immersive video system is fairly 
powerful . It . . . 

...or several such computers that are linked together if video must be 
processed in real time --i.e., as television. Especially if the immersive 
video is real time — i.e., as television — then some or all of the 
computers normally incorporate hardware graphics... 

... an immersive video system are both general purpose and conventional but 
are, at the present time (circa 1995) typically "state-of-the-art", and 
of considerable cost ranging to tens, and... in the teeth", it is very 
difficult for the system (especially initially, and in real time as 
television) , to process through a discordant scene occurrence, such as the 
stadium parachutist, so. . .process a completely amorphous unstructured video 
scene— the windy jungle — into 3D, especially in real time (i.e., as 
television) . It is, however, eminently possible to process many scenes of 
great... are, dealt with. The typically greater portion of any scene that is 
(at • any one time ) static is neither processed nor re-processed from 
moment to moment, and from frame to . . . to present, as a selected stream of 
video data containing a selected view, first scan time video data and 
second scan time video data that is displaced, each relative to the 
other, in accordance with the location... 

. . . stream is presented in a one color, or, more commonly at present, at a 
one time or in a one polarization, while the other video stream is 
presented in a separate color, or at a separate time , or in an 
orthogonal polarization, and each stream is separately gated to the eye (at 
greater than the eye flicker fusion frequency=70 Hz) by action of colored 
glasses, or time -gated filters, or polarizing filters, then the image 
presented to the eyes will appear to. . .present invention provides the vast, 
relatively inexpensive, "database" of the real world (at all scales, time 
compressions/expansions, etc.) as a suitable "field of operation" (or 
"playground") for the characters of... a three-dimensional video 
representation of everything that s within a scene, especially in real 
time (i.e., as television). 

Any desired scene view is then synthesized (alternatively, "extracted") 
from the... be rendered stereoscopically, as desired. 

The synthesized/extracted video views may be processed in real time , as 
television. 

Any and all synthesized/extracted video views contain only as much 
information as... 

the term "virtual world" is becoming a misnomer. These applications, 
which require accurate and real-time modeling of actions and events in 
the "real world" (e.g., gravity), interact with a . . . and experiences that 
are most valuable, entertaining and pleasing to most people most of the 
time are sensations and experiences of the real world, or at least 
sensations and experiences that... See B. Chapin, Telepresence Definitions, 
a World Wide Web (WWW) document on the Internet at URL 
http://cdr.stanford.edu/html/telepresence/definition.html, 1995. Key 

features of telepresence applications are: 1) the entire application is 
real- time ; 2) the virtual world is reasonably faithful to the real world 
being mimicked; 3) since real-time and real-world are cardinal, sensors 
should be used in acquiring the virtual world in... 
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. . . implementation coupled with expectation-driven, need-based analysis 
(described in Section 5) ensures near real-time model construction. The 
preferred immersive video system, described in Section 4, reconstructs 
realistic monocular and stereo vistas from the viewer perspective (see, for 
example, FIG. 10b) . 

Even in non-real time applications, video-based systems, such as the 
one taught in this specification, can be very . . . Assimilator and used to 
construct a comprehensive representation of events occurring in the scene 
over time (e.g. object movements and positions). 

The Assimilator thus models spatial-temporal activities of objects... in a 
courtyard are dynamic objects pillars standing in the courtyard are static 
objects. The time variance of the set O sub i (t) is a result of the 
time variation of the dynamic objects. 

As befit their name, static objects do not vary with time . The set of 
values of these objects at any instant comprises the state of the... In this 
case, comparison of objects in D sub v, i (t) at a particular time instant 
t with objects in S sub o,i can help anticipate and resolve such... 
realistic assumptions about objects of interest: 

First, these objects are in motion most of the time . 

Second, these objects move on known planar surfaces. 

Third, these objects are visible from at ... capability to integrate these 
techniques into analysis and assimilation modules when they become 
available. To date , evaluation of the preferred MPI-Video system has been 
done only by use of f ixed ... system is to keep track of object behavior and 
events unfolding in the environment over time . 

3.3 Best View Selection 

The View Selector can use a variety of criteria and. . . 

... sub i where the index i epsilon {1, . . . , N) varies over all cameras. 
At every time step, t, each camera produces a video frame, F sub,i,t. The 



...i sub BV is the index of the camera which produces the best view at 
time t. Then, the best view is found by selecting the frame from camera C 
sub iBV at time t, i.e., the best view is F sub iBV, t. 

Some possible ...object orientation. 

In the case of a least occluded view criteria, the system chooses, at 
time t, that frame from the camera in which an object of interest is 
least occluded. .. for C sub i, e.g. as a weighted linear sum. The use of the 
time t in this equation supports a best view optimization which uses a 
temporal selection criteria involving many frames over time , as well as 
spatial metrics computed on each frame. This is addressed in the following 
...which chooses the optimum from the set of g sub i,t 's. Note that time 
does not appear explicitly in the right hand side of this equation, 
indicating that the same best view evaluation is applied at each time 
step t. Note, in this case, the same g (here, a weighted linear sum) is... 
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...is possible. The best view is a frame from a particular camera. However, 
smoothness over time may also be important to the viewer or a system 
processing module. Thus, ...is an idea that has captured the imagination of 
science fiction writers for a long time . Although not feasible in its 
entirety, it is conjectured that limited telepresence will play a... See M. 
Hansen, P. Anandan, K. Dana, G. van der Wal, and P. Burt, Real-time Scene 
Stabilization and Mosaic Construction, in ARPA Image Understanding 
Workshop, Monterey, Calif., Nov. 13-16... See M. Hansen, P. Anandan, K. 
Dana, and G. van der Wal et al., Real-time scene stabilization and mosaic 
construction, in Proc. of Workshop on Applications of Computer Vision, 
pages... FIG. 1 outlines the steps involved. Algorithm 1 is the vista 
compositing algorithm. At each time instant, multiple vistas are computed 
using the current dynamic model and video streams from multiple ... the 
relative placements of all four cameras. Frames from four cameras (for the 
same arbitrary time instant, 00:21:08:02) are shown in FIG. 6. The scene 
contains three walkers ... effect is more discernible. A bird's eye view of 
the walkway for the same time instant is shown in FIG. 8a. Again, white 
portions represent areas not covered by any. . .model, the occupancy of only 
those voxels whose state could have changed from the previous time 
instant is continuously determined. Using higher level knowledge, and 
information from prior processing, this computation ... in height. 

Second, bounds are put on where the objects may be at the current time 
instant based on prior state, tracking information, assumptions about 
surfaces of motion etc. 

The former... Challenge) controls these video processing work stations 
(slaves) and maintains the Environment Model (and associated temporal 
database ) . The central master and the remote slaves communicate at a high 
symbolic level and minimal... in interaction and immersion for diverse 
virtual world applications. One of these application is real-time virtual 
video, or virtual television, or telepresence — next discussed in the 
following section 6. Various... 6. Immersive Telepresence 

Immersive telepresence, or visual reality, is an immersive, interactive 
and realistic real- time rendition of real-world events captured by 
multiple video cameras placed at different locations in the environment. It 
is the real-time rendition of virtual video; "virtual television" instead 
of just "virtual video". 

Unlike virtual reality, which ... Corporation . 

It also impossible to convey in the drawings when something is happening 
in real time . In some cases the multiple video feeds from a scene that 
was processed in real time to present telepresence to a user/view were 
also recorded and were then later processed telepresence and immersive 
video are these. First, more computer processing time is clearly 
available in non-real time immersive video than in immersive 
telepresence. This may not be, however, of any great significance ... real, 
remote scenes. This specification and this section instead describe 
immersive telepresence, being the real-time interactive and realistic 
rendition of real-world ...See M. Hansen, P. Anandan, K. Dana, G. van der 
Wal, and P. Burt, Real-time scene stabilization and mosaic construction, 
in ARPA Image Understanding Workshop, Monterey, Calif., Nov. 13-16... See M. 
Hansen, P. Anandan, K. Dana, and G. van der Wal et al . , Real-time scene 
stabilization and mosaic construction, in Proc. of Workshop on Applications 
of Computer Vision, pages... and present results for the Sam campus walkway 
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covered by multiple video cameras — only this time as television in real 
time as opposed to non-real-time video. Future directions for VisR are 
outlined in section 6.4. 

6.2 Visual Reality... interactive viewer can Valk through^ this dynamic, 
live environment in as it exists in real time (e.g., as seen through 
television) . 

6.3.1 Building a Comprehensive, Dynamic 3-D... 3-D position of each dynamic 
object detected by a motion segmentation module in real time . A priori 
information about the scene and camera calibration parameters, coupled with 
the assumption that... to the viewing position is used in the rendition. 
This is then repeated for every time instant (or every frame) assuming 
stationary view-port. To generate a "fly-by" sequence this... and zooming 
into the model is expected to be useful in representing sporting events. To 
date no problems with camera jitter, frame dropouts etc. have been 
encountered in the prototype system. . .N. Negroponte, Being digital, Knopf, 
New York, 1995. 

8. Immersive Video/Television At the Present Time , or How to Use Five 
Hundred Television Channels Beneficially 

The diverse sophisticated video presentations discussed in his 
specification are so discussed in the necessarily formative terminology of 
the present time , when not enough people have seen these effects of these 
video presentations so as to ... acknowledging that they cannot perfectly 
foresee the future. 

Immersive video may be divided into real-time applications, i.e., 
immersive television, and all other, non-real-time , applications where 
there is, mercifully, more time to process video of a scene. 

Both applications are presently at developed to a usable... 

. . . entertainments constitute a separate market from computer games and 
interactive computerized tutors at the present time . 

3.1 Monitoring Live Events in Real Time or Near Real Time 

With high speed video digitalizers, an immersive video system based on a 
single engineering work station class computer can, at the present time , 
process and monitor (being two separate things) the video of live events in 
real time or near real time . 

Such a system can, for example, perform the function of a "television 
sports ... it may ultimately be possible to have television views on 
demand. 

Another presently-realizable real-time application is security, as at, 
for example, airports. An immersive video system can be directed ... video of 
an entire observed head is available -for comparison. 

The ultimate use of real-time and near-real-time immersive video may 
in Fact be in machine perception as opposed to human entertainment. The... 
for the entertainment or education of humans. 
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Meanwhile to developments in immersive television, the processing of 
video information--which is not required to transpire in real time --and 
the communication of video inf ormation--which may be by disc or like 
transportable storage... with considerable computer processing. A complete 
3D database of fine detail can be developed, over time and by con^uter 
processing, from historical multiple video feeds of anything from a 
football game. . . 

When recorded, a scene from the 3D database can be "played back" at 
normal, real- time , speeds and in accordance with the particular desires 
of a particular end viewer/user by. . . 

... the receiving, the finding, the projecting, the testing and the 
evaluating for an instance of time of each video frame assuming a 
stationary viewpoint. 

16. The hypermosaicing composing method according to... perspective 
received from the viewer. 

19. The method according to claim 17 performed in real time as 
television presented to a viewer interactively in accordance with the 
viewer-specified criterion. 

20... wherein the generating of the selected two-dimensional stereoscopic 
virtual video image is in real time on demand, thus interactive virtual 
television . 

36. A computerized method for presenting video images including ... streams 
transpires by tracking changes in scene element representations in the 
multiple video streams over time . 

41. The method according to claim 36 wherein the environmental model 
determines whether any scene... 

...is of a varying, and not a same and consistent, view on the scene from 
time to time . 
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